Electronic device and method of controlling thereof

ABSTRACT

Disclosed is an electronic device. The electronic device may execute an application for transmitting and receiving at least one of text data or voice data with another electronic device using the communication module, in response to occurrence of at least one event, based on receiving at least one of text data or voice data from the another electronic device, identify that a confirmation is necessary using the digital assistant based on at least one of text data or voice data being generated based on a characteristic of ab utterance using a digital assistant, generate a notification to request confirmation using the digital assistant based on confirmation being necessary, and output the notification using the application. 
     A method for identifying that a confirmation is necessary may include identifying using voice data or text data that is received from another electronic device using a rule-based or AI algorithm. 
     When a confirmation is necessary is identified using the AI algorithm, the method may use machine learning, neural network, or a deep learning algorithm.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2018-0153683, filed on Dec. 3, 2018,in the Korean Intellectual Property Office, the disclosure of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to an electronic device that transmits andreceives text data or voice data with another electronic device using adigital assistant.

2. Description of Related Art

In addition to a traditional input method using a keyboard or a mouse,recent electronic devices may support various input methods such as avoice input. For example, an electronic device such as a smartphone or atablet may recognize a user's voice input while a digital assistantfunction is executed, execute an operation corresponding to the voiceinput, or provide a search result.

The digital assistant function has been developed based on technologyfor processing a natural language. The technology for processing anatural language is for grasping an intention of a user's utterance andproviding a user with a result corresponding to the intention.

In this case, the electronic device may provide a result using a displayor provide a result by a voice.

SUMMARY

Embodiments of the disclosure address at least the above-mentionedproblems and/or disadvantages and provide at least the advantagesdescribed below. Accordingly, an example aspect of the disclosure is toprovide a method for identifying a situation where a user's confirmationis necessary when a digital assistant generates voice data or text datausing a characteristic of user's utterance.

According to another example aspect of the disclosure, when the digitalassistant generates voice data or text data using a characteristic ofuser's utterance, a method of outputting a display that the data isgenerated by the digital assistant is provided.

According to another example aspect of the disclosure, a method ofgenerating and outputting new text data or voice data using at least apart of voice data or text that is received or transmitted with anotherelectronic device by the digital assistant is provided.

According to an example embodiment, an electronic device includes acommunication module comprising communication circuitry, a processor,and a memory, and the memory may store at least one command which, whenexecuted by the processor, controls the electronic device to: execute anapplication for transmitting and receiving at least one of text data orvoice data with another electronic device using the communication modulein response to occurrence of at least one event, identify—whetherconfirmation is necessary using the digital assistant based on receivingat least one of text data or voice data from the another electronicdevice and based on at least one of text data or voice data beinggenerated based on a characteristic of an utterance using a digitalassistant, generate a notification requesting confirmation using thedigital assistant based on the confirmation being necessary, and outputthe notification using the application.

According to an example embodiment, based on the electronic devicegenerating voice data or text data based on a characteristic anutterance using the digital assistant, a situation where confirmation isnecessary may be identified.

According to an example embodiment, based on the electronic devicegenerating voice data or text data based on a characteristic of anutterance using the digital assistant, a display notifying that the datais generated by the digital assistant may be output.

According to an embodiment, the electronic device may generate andoutput new text data or voice data using at least a part of voice dataor text data that is transmitted or received with another electronicdevice using the digital assistant.

Other aspects, advantages, and salient features of the disclosure willbecome apparent to those skilled in the art from the following detaileddescription, which, taken in conjunction with the annexed drawings,discloses various embodiments of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a diagram illustrating an example of sharing a text messagewith another electronic device using a digital assistant functionaccording to an embodiment of the disclosure;

FIG. 2 is a block diagram illustrating an example electronic device anda server according to an embodiment;

FIG. 3 is a diagram illustrating an example in which an electronicdevice identifies and responds to a case where an opinion of a user isnot consistent with an opinion of a third party using a digitalassistant according to an embodiment;

FIG. 4 is a diagram illustrating an example in which an electronicdevice corresponds to a dialogue with a third party using a reply speedhistory of a user using a digital assistant according to an embodiment;

FIG. 5 is a diagram illustrating an example in which an electronicdevice confirms new information and responds to a dialogue with a thirdparty using a digital assistant according to an embodiment;

FIG. 6 is a diagram illustrating an example of summarizing anddisplaying a dialogue content by an electronic device using a digitalassistant according to an embodiment;

FIG. 7 is a diagram illustrating an example of summarizing anddisplaying a dialogue content by an electronic device using a digitalassistant according to various embodiments;

FIG. 8 is a diagram illustrating an example of guiding by an electronicdevice to a user of another electronic device while waiting forconfirmation of a user using a digital assistant according to anembodiment;

FIG. 9 is a block diagram illustrating an example process of executing adigital assistant according to an embodiment;

FIG. 10 is a diagram illustrating an example learning model used fordetermining level of importance to generate a notification of a user bya digital assistant according to an embodiment;

FIG. 11 is a diagram illustrating an example function of a userinstruction confirmation model according to an embodiment;

FIG. 12 is a diagram illustrating an example function of a reply typeconfirmation model according to an embodiment;

FIG. 13 is a diagram illustrating an example function of a reply timeconfirmation model according to an embodiment;

FIG. 14 is a diagram illustrating an example function of a reply timeconfirmation model according to an embodiment;

FIG. 15 is a flowchart illustrating an example operation of transmittingand receiving text data or voice data with a third party by anelectronic device using a digital assistant according to an embodiment;

FIG. 16 is a flowchart illustrating an example operation of transmittingand receiving text data or voice data with a third party using a digitalassistant by an electronic device and a server according to anembodiment; and

FIG. 17 is a block diagram illustrating an example electronic device ina network environment according to various embodiments.

The same reference numerals are used to represent the same elementsthroughout the drawings.

DETAILED DESCRIPTION

Hereinafter, various example embodiments of the disclosure will bedescribed with reference to the accompanying drawings. However, it is tobe understood that the disclosure is not limited to specificembodiments, but includes various modifications, equivalents, and/oralternatives according to embodiments of the disclosure. Throughout theaccompanying drawings, similar components will be denoted by similarreference numerals.

The terms “have”, “may have”, “include”, and “may include” used in theexample embodiments of the present disclosure indicate the presence ofcorresponding features (for example, elements such as numerical values,functions, operations, or parts), and do not preclude the presence ofadditional features.

In the description, the term “A or B”, “at least one of A or/and B”, or“one or more of A or/and B” may include all possible combinations of theitems that are enumerated together. For example, the term “at least oneof A or/and B” includes (1) including at least one A, (2) including atleast one B, or (3) including both at least one A and at least one B.

In addition, expressions “first”, “second”, or the like, used in thedisclosure may indicate various components regardless of a sequenceand/or importance of the components, may be used to distinguish onecomponent from the other components, and do not limit the correspondingcomponents. For example, a first user device and a second user devicemay indicate different user devices regardless of a sequence orimportance thereof. For example, the first component may be named thesecond component and the second component may also be similarly namedthe first component, without departing from the scope of the disclosure.

The term such as “module,” “unit,” “part”, and so on may be used torefer to an element that performs at least one function or operation,and such element may be implemented as hardware or software, or acombination of hardware and software. Further, except for when each of aplurality of “modules”, “units”, “parts”, and the like needs to berealized in an individual hardware, the components may be integrated inat least one module or chip and be realized in at least one processor.

When any component (for example, a first component) is (operatively orcommunicatively) coupled with/to or is connected to another component(for example, a second component), it is to be understood that anycomponent may be directly coupled with/to another component or may becoupled with/to another component through the other component (forexample, a third component). On the other hand, when any component (forexample, a first component) is “directly coupled with/to” or “directlyconnected to” to another component (for example, a second component), itis to be understood that the other component (for example, a thirdcomponent) is not present between the directly coupled components.

Terms used in the disclosure may be used to describe specificembodiments rather than restricting the scope of other embodiments.Singular forms are intended to include plural forms unless the contextclearly indicates otherwise. Terms used in the disclosure includingtechnical and scientific terms may have the same meanings as those thatare generally understood by those skilled in the art to which thedisclosure pertains. Terms defined in a general dictionary among termsused in the disclosure may be interpreted as meanings that are the sameas or similar to meanings within a context of the related art, and arenot interpreted as ideal or excessively formal meanings unless clearlydefined in the disclosure. In some cases, terms may not be interpretedto exclude embodiments of the disclosure even where they may be definedin the disclosure.

FIG. 1 is a diagram illustrating an example of sharing a text messagewith another electronic device using a digital assistant functionaccording to an embodiment.

The digital assistant function may refer, for example, to receiving andrecognizing a user's voice or a text input by the user, and generating acommand for operating an application in response to the user's voice orthe text input by the user. Operating an application may refer, forexample, to generating and outputting an answer in a form of voice ortext in response to a user input. Operating an application may refer,for example, to performing at least one or more functions in response toa user input. However, the digital assistant function is not limitedthereto.

According to various embodiments, the digital assistant function mayrefer, for example, to generating a command to operate an application inresponse to the voice data or text data received from another electronicdevice.

According to an example embodiment, a digital assistant function(hereinafter referred to as a digital assistant) may be implementedusing an electronic device 10. The digital assistant may be implementedusing the electronic device 10 and at least one server. The server mayinclude a cloud server.

According to an example embodiment, the electronic device 10 may providea function or a service that is necessary for a user through anapplication (or application program) (for example, a phone callapplication, a message application, a social network service (SNS)application) stored in the electronic device 10.

According to an embodiment, the electronic device 10 may include, forexample, and without limitation, a mobile phone, a smartphone, apersonal digital assistant (PDA), a mobile robot, an artificialintelligence (AI) speaker, various devices connectible to Internet suchas a notebook computer, or the like.

Hereinafter, a situation in which the digital assistant is executed isdescribed.

According to an embodiment, the electronic device 10 may receive auser's utterance as a user input. The electronic device 10 may receive auser's utterance and operate an application based on the user'sutterance.

According to an embodiment, the electronic device 10 may implement adigital assistant in association with at least one server.

For example, the electronic device 10 may receive the user's utteranceas a user input. The electronic device 10 may transmit the receivedutterance to at least one or more server. The at least one server mayreceive a user voice input from the electronic device 10 and change theuser voice input to text data. The at least one server may generate (orselect) a path rule based on text data. The path rule may includeinformation on an action (or operation) for performing a function of anapplication or information about the parameters necessary to execute theoperations. The path rule may also include an order of operation of theapplication. In addition, the path rule may include an order ofoperations of a plurality of applications. The electronic device 10 mayreceive the path rule, select the application according to the pathrule, and execute the operation included in the path rule in theselected application.

According to an embodiment, the at least one server may include adatabase in which user information is stored. The at least one servermay receive user information (for example, call logs, text inputhistory, context information, application execution information, or thelike) from the electronic device 10, and store the same in the database.The at least one server may use the information included in the databaseto generate a path rule for the user input.

According to an example embodiment, the at least one server may includea database in which information about functions stored in the electronicdevice 10, introduction of an application, or information on thefunctions to be provided are stored. For example, the at least oneserver may generate a database for functions that the user may use usingthe user information. The electronic device 10 may receive informationabout functions that may be provided from at least one server through acommunication network and provide the received information to the user.

According to an example embodiment, the electronic device 10 may receivethe user input as a text. For example, the electronic device 10 and theat least one server may perform the same operations as those describedabove, except for the operation of converting the user's utterance intotext, and execute the operations included in the path rule.

According to an example embodiment, the electronic device 10 may performthe same or similar operations of the at least one server describedabove. For example, the electronic device 10 may receive the user'sutterance and convert the user's utterance into text data using thedigital assistant. The electronic device 10 may generate the path rulebased on the text data. The electronic device 10 may select anapplication according to the path rule, and execute an operationincluded in the path rule in the selected application.

According to various embodiments, the at least one server or theelectronic device 10 may include a learning model that enhancesrecognition capabilities through learning, such as a natural languageunderstanding module, a natural language generation module, or the like.For example, the learning model may be a model trained using AIalgorithms, such as, for example, and without limitation, machinelearning, neural networks, deep learning algorithms, or the like. Forexample, the AI algorithm may be used in the process of recognizing theuser's utterance and changing the utterance to the text data. Adescription thereof will be described in greater detail below withreference to FIGS. 9 and 10.

Referring to FIG. 1, the electronic device 10 may execute an applicationto receive or transmit at least one of the text data or voice data withanother electronic device, in response to occurrence of at least oneevent.

The at least one event may refer, for example, to receiving, from auser, a command executing an application that transmits and receives atleast one of text data or voice data with another electronic device. Theat least one event may refer to receiving text data or voice data fromanother electronic device.

The application which receives or transmits at least one of the textdata or voice data with another electronic device may be, for example,and without limitation, at least one of a message application, a voicecall application, a social network service (SNS) application, or thelike.

Hereinbelow, it is assumed and described that the electronic device 10executes the digital assistance using a message application and performsdialogue with another electronic device. However, it will be understoodthat this is simply for convenience and ease of explanation, and thatthe disclosure is not limited thereto.

According to an embodiment, the electronic device 10 may execute amessage application and display the text data generated in response tothe user input on the display 11 and an execution screen 110 of themessage application displaying text data received from anotherelectronic device.

Referring to FIG. 1, the electronic device 10 may display a message “Doyou have time today?” 121 received from another electronic device on theexecution screen 110 of the message application. Information (forexample, an image K of a counterpart) related to the counterpart usingthe message application via another electronic device may be displayedon the execution screen 110 of the message application.

According to an embodiment, when at least one of text data or voice datais received from another electronic device, the electronic device 10 maygenerate at least one of text data or voice data based on the user'sutterance characteristic using a digital assistant and output the datausing the application.

The electronic device 10 may generate the path rule based on the textdata or voice data received from another electronic device using thedigital assistant. For example, the electronic device 10 may generatethe path rule to generate a reply, in response to “Do you have timetoday?” 121.

The electronic device 10 may generate the text data of “I have anappointment until 3 pm” 131 using the pre-stored characteristic of theuser's utterance, transmit the text data to another electronic device,and display on the execution screen 110 of the message application. Forexample, the electronic device 10 may obtain the characteristic of theuser's utterance using at least one of the call log, test input historyof the user, and generate the text data to which the characteristic ofthe user's utterance of a user who does not correctly pronounce the endof the speech is reflected. The electronic device 10 may displayinformation related to the user (for example, an image J of the user) onthe execution screen 110 of the message application.

The electronic device 10 may generate a text using the user's contextinformation (for example: location, time, etc.), application executioninformation (for example: web search history, information onapplications executed a lot, information on a recently installedapplication, etc.). For example, the electronic device 10 may generate asentence “I think A movie is interesting” 133 with respect to the textdata received from another electronic device of “Do you want to watchany movie?” 123 using the search history of the user who searched Amovie.

According to various embodiments, the electronic device 10 may transmit,to another electronic device, an indicator 131 a indicating that thetext data corresponding to the text received from another electronicdevice is generated and displayed using the digital assistant, alongwith the generated text data, and may display the indicator 131 a on theexecution screen 110 of the message application.

According to an embodiment, the electronic device 10 may identify a casewhere a user's confirmation is necessary using the digital assistance ina state where at least one of the text data or voice data is generatedbased on the characteristic of the user's utterance.

When the user's confirmation is necessary may be a case whereconfirmation on the user's instruction is necessary. For example, thismay be a case in which some content included in the text data or voicedata generated by the user is not the same as some content of the textdata or voice data received from another electronic device.

The case where the user's confirmation is required may be, for example,a case that the text data or the voice data that the digital assistantshould generate needs to determine whether to accept a proposal. Forexample, if a proposal is included in the text data or voice datareceived from the another electronic device, the electronic device 10may identify this case as the case where a confirmation of the user isrequired, since a prudent reply may be necessary.

The case where the user's confirmation is necessary may be a case, forexample, where the user has a history that the user utters a voicesimilar to the voice data that the digital assistant should generate,and confirmation of the user's utterance response time is necessary. Inother words, if the user's time corresponding to the other party's voiceis longer than a predetermined time (for example, more than the user'saverage reply ownership time), it may be a situation that a prudentreply is required, so that the electronic device 10 may identify thiscase as the case that the confirmation from the user is necessary.

The case where the user's confirmation is necessary may be, for example,a case where similarity of at least some content of the text data orvoice data received from another electronic device and text data orvoice data prestored in the electronic device is less than or equal to apredetermined value. This case may be a case requiring a prudent replyand thus, the electronic device 10 may identify the case as a caserequiring the user's confirmation.

Referring to FIG. 1, the electronic device 10 may display “Then, let'swatch the movie at 4 pm and let's go for dinner” 125 on the executionscreen 110 of the message application.

For example, the electronic device 10 may identify that the proposal ofthe counterpart is included in the content of the received message. Theelectronic device 10 may identify that the user's confirmation isnecessary in a situation that the text data is generated in response to“Then, let's watch the movie at 4 pm and let's go for dinner” 125received from the another electronic device, based on the identifiedresult.

According to an embodiment, the electronic device 10 may, while theuser's confirmation is being necessary, generate a notificationrequesting the user's confirmation using the digital assistant, andoutput the notification using the application.

Referring to FIG. 1, the electronic device 10 may generate anotification requesting the user's confirmation in response to “Then,let's watch the movie at 4 pm and let's go for dinner” 125 received bythe another electronic device using the digital assistant.

The notification requesting confirmation to the user may include, forexample, based on the text data received from the another electronicdevice, the text data for the appointment at 4 pm and a selection objectthat may receive the user's input regarding the text data. Referring toFIG. 1, the electronic device 10, using the digital assistant, maygenerate the text data “Shall we make a schedule at 4 pm?” 135, anobject 135 b that receives confirmation, an object 135 c that receivesrejection, and an object 135 d that receives a user's direct inputprogress, and display the same on the execution screen 110 of a messageapplication. In this case, the electronic device 10 may display the textdata regarding the 4 pm appointment along with the indicator 135 aindicating that the selected object that is capable of receiving theuser's input has not been transmitted to the another electronic device.

Referring to FIG. 1, the electronic device 10 may transmit the user'sdirect input “OK. Let's see then” 127 that is directly input by the userto the another device, and display the same on the execution screen 110of the message application. For example, the electronic device 10 maydisplay a pop-up window for inputting a text, in response to a user'sinput of selecting the object 135 d for receiving the direct inputprogress. The electronic device 10 may transmit the input text data toanother electronic device through a popup window for inputting text anddisplay the text data on the execution screen 110 of the messageapplication.

According to various embodiments, the electronic device 10 may generateand display the text data corresponding to the text received from theanother electronic device using at least one server. For example, theelectronic device 10 may transmit the text data received from anotherelectronic device to at least one server, receive the text data that isgenerated by at least one server based on the received text data, anddisplay the text data on the execution screen 110 of the messageapplication.

According to various embodiments, the electronic device 10 may registera schedule based on the content of text data or voice data that isreceived or transmitted from or to the another electronic device usingthe digital assistant. For example, the electronic device 10 may add aschedule to visit B hair salon at 4 pm to the schedule application usingthe digital assistant and notify the user of adding the schedule byexecuting the schedule application.

According to various embodiments, the electronic device 10 may provide anotification to a user using various methods when user's confirmation isrequired. For example, the electronic device 10 may provide a userinterface for selecting an application for providing a notification to auser by the digital assistant. For example, the electronic device 10 mayguide to select at least one of a call application, an e-mailapplication, an SMS application, and a message application. In addition,the electronic device 10 may guide to use all of the above-describedapplications or select the priority of the operation with respect to theaforementioned application.

As described above, according to an example embodiment, the electronicdevice 10 may generate the text data or voice data in response to thetext data or voice data received from the another electronic deviceusing the digital assistant, in place of the user. For example, theelectronic device 10 may identify a situation where the user'sconfirmation is necessary using the digital assistant and provide theuser of a notification. When a prudent decision is necessary, the usermay actively intervene in the dialogue between the digital assistant anda third party.

FIG. 2 is a block diagram illustrating an example electronic device andserver according to an embodiment.

Referring to 2-a of FIG. 2, the electronic device 10 may include aprocessor (e.g., including processing circuitry) 210, a communicationmodule (e.g., including communication circuitry) 220, and a memory 230,but is not limited thereto. The electronic device 10 may omit a part ofthe components or include additional components.

According to an embodiment, the processor 210 may execute a programstored in the memory 230 by controlling the memory 230 and retrieve orstore necessary information.

For example, the processor 210, may include various processing circuitryand in response to at least one event occurrence, may execute anapplication for transmitting or receiving at least one of the text dataor voice data with another electronic device using the communicationmodule, and when at least one of the text data or voice data is receivedfrom the another electronic device, generate at least one of the textdata or voice data based on the characteristic of the user's utteranceusing the digital assistant, output the data using the application, andin the situation where at least one of the text data or voice data isgenerated based on the characteristic of the user's utterance, identifya situation where the user's confirmation is necessary using the digitalassistant, and in the situation where the user's confirmation isnecessary, generate a notification requesting the user of confirmationusing the digital assistant, and output the notification using theapplication.

According to an embodiment, the communication module 220 may includevarious communication circuitry and connect the electronic device 10with an external device (for example: another electronic device, aserver, or the like) under the control of the processor 210. Thecommunication module 220 may include various components includingvarious communication circuitry which implement various wired orwireless communication methods such as, for example, and withoutlimitation, wireless local area network (LAN), Bluetooth, wiredEthernet, or the like, to correspond to the function and structure ofthe electronic device 10.

According to an embodiment, the memory 230 may store an application fortransmitting and receiving at least one of text data or voice data withanother electronic device using the communication module in response toat least one event occurrence. When at least one of the text data or thevoice data is received from another electronic device, using the digitalassistant, at least one of the text data or the voice data may begenerated based on the characteristic of user's utterance, and the datamay be output using the application. In a situation in which at leastone of text data or voice data is generated based on the characteristicof the user's utterance, a situation where the user's confirmation isnecessary is identified using the digital assistant, and in thesituation where the user's confirmation is necessary, a notificationrequesting the user's confirmation may be generated using the digitalassistant, and commands that are set to output the application may bestored.

Referring to 2-b of FIG. 2, a server 20 may include a data acquisitionunit (e.g., including data acquisition circuitry) 250, a data processor(e.g., including data processing circuitry) 260, and a data outputter(e.g., including data output circuitry) 270. The server 20 may includeat least one server as illustrated in FIG. 1.

According to an embodiment, the data acquisition unit 250 may includevarious data acquisition circuitry, such as, for example, and withoutlimitation various processing circuitry and/or executable programelements, and acquire at least one of the text data or voice data froman external device.

According to an embodiment, the data processor 260 may include variousdata processing circuitry, such as, for example, and without limitation,various processing circuitry and/or executable program elements, and,when at least one of the text data or voice data is received from theexternal device, may generate at least one of the text data or voicedata based on the characteristic of the user's utterance using thedigital assistant. The data processor 260, while at least one of thetext data or voice data is generated, may identify the situation wherethe user's confirmation is necessary using the digital assistant.

According to an embodiment, the data outputter 270 may include variousdata outputting circuitry and transmit the generated text data or voicedata to an external device. The data outputter 270, in a situation wherethe user's confirmation is necessary, may transmit information on thesituation where the user's confirmation is necessary, to the externaldevice.

FIG. 3 is a diagram illustrating an example in which an electronicdevice identifies and responds to a case where an opinion of a user isnot consistent with an opinion of a third party using a digitalassistant according to an embodiment.

Referring to 3-a of FIG. 3, the electronic device 10 may receive theutterance of the user J. For example, the electronic device 10 mayreceive the utterance of the user J of “please make a reservation of Bhair salon at 4 pm” 310. The electronic device 10 may convert thereceived utterance to the text data.

According to an embodiment, the electronic device 10 may generate thepath rule based on the text data using the digital assistant. Theelectronic device 10 may select an application according to the pathrule, and execute an operation included in the selected application.

For example, the electronic device 10 may perform an operation ofgenerating and outputting a notification informing that the utterance ofthe user J has been received using the digital assistant. In addition,the electronic device 10 may execute an operation of making areservation by calling B hair salon. For example, the electronic device10 may generate a voice data, “Yes, I see. B hair salon phone number is02-123-4567” 312 using the digital assistant and output the same using amicrophone (not shown). The electronic device 10 may generate text data“Yes, I see. B hair salon phone number is 02-123-4567” 312 using thedigital assistant, and output the text data using the messageapplication execution screen.

In FIG. 3, 3-b illustrates an example in which the electronic device 10makes a call according to the generated path rule.

According to an embodiment, the electronic device 10 may execute a callapplication to make a phone call to B hair salon using the digitalassistant in reference numeral 3-a of FIG. 3, and generate a path ruleto make a reservation by generating voice data in response to thedialogue of the counterpart.

Referring to 3-b of FIG. 3, the electronic device 10 may start a callwith another electronic device 300 by executing a call application. Theelectronic device 10 may receive voice data “This is B hair salon” 314through the call application. In response to the received voice, theelectronic device 10 may generate voice data, “Can I make a reservationat 3 pm today?” 316, and transmit the voice data to the anotherelectronic device 300 with which call is connected using the callapplication.

The electronic device 10 may receive the voice data “Oh . . .reservation is full at 3 pm. How about 4 pm?” 318 through the callapplication.

According to an embodiment, the electronic device 10 may identify thatthe received voice includes a content of requesting 4 pm that isdifferent from the reservation time 3 pm included in the voice that isgenerated and transmitted previously.

Based on that the reservation time requested by the user J and thereservation time included in the voice received from another electronicdevice 300 is different, the electronic device 10 may generate andoutput a notification requesting confirmation of the user J using thedigital assistant. In this example, the electronic device 10 maygenerate and transmit the voice “Wait a minute” 320 to the anotherelectronic device 300 using the digital assistant, so that thecounterpart using the another electronic device 300 can wait.

Referring to 3-c of FIG. 3, the electronic device 10 may generate anotification requesting the user's confirmation using the digitalassistant.

The notification for requesting confirmation to the user J may be, forexample, voice data including the content to change the reservation timewhich was 3 pm to 4 pm, based on the voice data received by the anotherelectronic device 300. According to various embodiments, the electronicdevice 10 may generate a notification requesting user's confirmationwith text data. In this example, the notification requesting user'sconfirmation may be text data 322 including a change in the reservationtime from 3 pm to 4 pm, an object 322 a for approving the change in thereservation time, an object 322 b for rejecting the reservation time,and an object 322 c that directly performs a call. The electronic device10 may generate a notification requesting user's confirmation anddisplay the notification on the execution screen of the messageapplication.

According to an embodiment, when the user's voice “Yes, please change”324 to approve the change in the reservation time is received, theelectronic device 10 may generate a voice to proceed the change in thereservation time using the digital assistant, and transmit the voice tothe another electronic device 300 using the call application.

As such, according to an embodiment, when some content of the user'srequest is different from some content included in the voice datareceived from the another electronic device, the electronic device 10may output a notification requesting the user for confirmation.

FIG. 4 is a diagram illustrating an example in which an electronicdevice corresponds to a dialogue with a third party using a reply speedhistory of a user using a digital assistant according to an embodiment.

Referring to 4-a of FIG. 4, the electronic device 10 may receiveutterance of the user J. For example, the electronic device 10 mayreceive utterance of the user J, “please make a reservation of B hairsalon at 3 pm” 410. The electronic device 10 may convert the receivedutterance to the text data.

According to an embodiment, the electronic device 10 may generate thepath rule based on the text data using the digital assistant. Theelectronic device 10 may select an application according to the pathrule, and execute an operation included in the selected application.

For example, the electronic device 10 may perform an operation ofgenerating and outputting a notification guiding that the utterance ofthe user J has been received using the digital assistant. In addition,the electronic device 10 may execute an operation of making areservation by calling B hair salon. For example, the electronic device10 may generate a voice data “Yes, I see. B hair salon phone number is02-123-4567” 412 using the digital assistant, and output the voice datausing a microphone (not shown). The electronic device 10 may generatetext data, “Yes, I see. B hair salon phone number is 02-123-4567” 412using the digital assistant, and output the text data using the messageapplication execution screen.

In FIG. 4, 4-b illustrates an example of making a call according to thegenerated path rule.

According to an embodiment, the electronic device 10 may execute a callapplication to make a call to B hair salon using the digital assistantin 4-a of FIG. 4, generate the voice data in response to the dialogue ofthe counterpart, and generate the path rule to proceed reservation.

Referring to 4-b of FIG. 4, the electronic device 10 may start a callwith another electronic device 400 by executing a call application. Theelectronic device 10 may receive voice data “This is B hair salon” 414through a call application. In response to the received voice, theelectronic device 10 may generate voice data, “Can I make a reservationat 3 pm today?” 416, and transmit the voice data to the anotherelectronic device 400 to which call is connected, using the callapplication.

The electronic device 10 may receive the voice data of “Yes, how do youlike your hair cut?” 418 through the call application.

According to an embodiment, the electronic device 10 may identify thetime which is required for making a reply by a user for received voice.

Based on the user J's reply after having spent more than a predeterminedtime in the past with respect to the voice data “Yes, how do you likeyour hair cut?”, the electronic device 10 may generate and output anotification requesting from the user J using the digital assistant. Inthis example, the electronic device 10 may generate and transmit a voice“wait a minute” 420 to the another electronic device 300 using thedigital assistant so as to induce the counterpart using the anotherelectronic device to wait.

Referring to 4-c of FIG. 4, the electronic device 10 may generate anotification for requesting the user's confirmation using the digitalassistant.

The notification requesting the user J of confirmation may be voice datanotifying that an inquiry is being made about a hair style based on thevoice data received by another electronic device 400, for example.According to various embodiments, the electronic device 10 may generatea notification for requesting the user's confirmation with text data. Inthis example, the notification for requesting the user's confirmationmay be text data 422 notifying that an inquiry is being made about ahair style. The electronic device 10 may generate a notificationrequesting the user's confirmation and display the notification on theexecution screen of the message application.

According to an embodiment, when the user's voice to determine the hairstyle of “sports hair” 424 is received, the electronic device 10 maygenerate a voice notifying the hair style and transmit the voice to theanother electronic device 400 using a call application.

According to an embodiment, the electronic device 10 may output anotification for requesting the user of confirmation using informationon a past reply speed of the user using the digital assistant.

FIG. 5 is a diagram illustrating an example in which an electronicdevice confirms new information and responds to a dialogue with a thirdparty using a digital assistant according to an embodiment.

Referring to 5-a of FIG. 5, the electronic device 10 may receive theutterance of the user J. For example, the electronic device 10 mayreceive the utterance of the user J of “Make a reservation of B hairsalon as usual” 510. The electronic device 10 may convert the receivedutterance to the text data.

According to an embodiment, the electronic device 10 may generate thepath rule based on the text data using the digital assistant. Theelectronic device 10 may select an application according to the pathrule, and execute an operation included in the selected application.

For example, the electronic device 10 may perform an operation ofgenerating and outputting a notification guiding that the utterance ofthe user J has been received using the digital assistant. In addition,the electronic device 10 may identify that the user J has a plurality ofhistories of making a reservation for X designer of B hair salon at 3pm. In addition, the electronic device 10 may execute an operation ofmaking a reservation by calling B hair salon.

For example, using the digital assistant, the electronic device 10 maygenerate the voice data, “Yes, I see. B hair salon phone number is02-123-4567” 512, and output the voice data using a microphone (notshown). The electronic device 10 may generate the text data “Yes, I see.B hair salon phone number is 02-123-4567” 512 and output the voice datausing the message application execution screen.

In FIG. 5, 5-b illustrates an example of making a call by the electronicdevice 10 according to the generated path rule.

According to an embodiment, the electronic device 10 may execute a callapplication to make a call to B hair salon using the digital assistantin 5-a of FIG. 5, generate the voice data in response to the dialog ofthe counterpart, and generate the path rule to make a reservation.

Referring to 5-b of FIG. 5, the electronic device 10 may start a callwith another electronic device 500 by executing a call application. Theelectronic device 10 may receive voice data “This is B hair salon” 514through a call application. In response to the received voice, theelectronic device 10 may generate voice data, “Can I make a reservationat 3 pm today?” 516, and transmit the voice data to the anotherelectronic device 500 to which a call is connected using the callapplication.

The electronic device 10 may receive the voice data of “Which designerwill you choose?” 518 from the another electronic device 500 through thecall application.

According to an embodiment, the electronic device 10 may identifywhether the received voice data has a new content, using the digitalassistant.

Based on that there is no obtained history, for the voice data “whichdesigner will you choose?” 518, the electronic device 10 may generateand output a notification requesting confirmation to the user J usingthe digital assistant. In this example, the electronic device 10 maygenerate and transmit a voice called “Wait a minute” 520 to the anotherelectronic device 500 using the digital assistant to induce thecounterpart using the another electronic device 500 to wait.

Referring to 5-c of FIG. 5, the electronic device 10 may generate anotification requesting the user's confirmation using the digitalassistant.

Notification for requesting confirmation to the user may be voice dataindicating that an inquiry is being made to the designer based on thevoice data received from the another electronic device 500, for example.According various embodiments, the electronic device 10 may generate anotification requesting user confirmation with text data. In this case,the notification requesting the user confirmation may be text data 522indicating that the designer is making an inquiry, objects 522 a forselecting a designer, and objects 522 b for directly making a call. Theelectronic device 10 may generate a notification requesting userconfirmation and display the notification on an execution screen of amessage application.

According to an embodiment, when a voice of the user J for requestingfinalization of a designer of “X designer” 524 is received, theelectronic device 10 may generate a voice for requesting finalization ofthe designer using the digital assistant, and transmit the voice to theanother electronic device 500 using the call application.

According to an embodiment, when a new content that has not been presentin the dialogue history is obtained using the digital assistant, theelectronic device 10 may output a notification for requesting the user'sconfirmation.

FIG. 6 is a diagram illustrating an example of summarizing anddisplaying a dialogue content by an electronic device using a digitalassistant according to an embodiment.

According to an embodiment, the electronic device 10 may generatesummary text data using a content included in text data transmitted andreceived with another electronic device using the digital assistant andprovide the same to the user. For example, the electronic device 10 maygenerate summary text data and provide the same to the user when theuser does not confirm at least some content of the text data transmittedand received by the digital assistant and the another electronic device.However, the situation in which the electronic device 10 providessummary text data using the digital assistant is not limited thereto.

According to an embodiment, the summary text data may include a name ofa counterpart using another electronic device. In addition, the summarytext data may include content that needs the user's confirmation. Forexample, the summary data may include at least one of a case where theconfirmation of the user's instruction is necessary, a case where aproposal is included in the text data received from the another device,a case where the text data which is the same as the case where the usermade a reply after spending a predetermined time on the utterancereceived from the other electronic device is received, and a case wherethe similarity between the text data or voice data received from anotherelectronic device and the text data or voice data previously stored inthe electronic device is less than or equal to a predetermined value.

Referring to FIG. 6, the electronic device 10 may generate a summarytext data, “A proposal to watch A movie at 4 pm is received during thedialogue with K” 610 using the digital assistant and display the data onthe execution screen 110 of the message application. In this case, theelectronic device 10 may display an entire view object 610 a thatdisplays all the contents included in the text data transmitted andreceived with another electronic device using the digital assistant.

According to an embodiment, the electronic device 10 may generate anddisplay text data requesting confirmation of a user using the digitalassistant.

Referring to FIG. 6, the electronic device 10 may generate text data of“Shall I make a reservation at 4 pm?” 612, an object 612 a receiving anapproval, an object 612 b that receives a rejection, and an object 612 cthat receives the user's direct input progress using the digitalassistant, and may display the same on the execution screen 110 of themessage application. In this example, the electronic device 10 maydisplay the text data for the 4 pm appointment and an indicator 612 dindicating that the selected object that is capable of receiving theuser's input has not been transmitted to the another electronic device.

According to an example embodiment, the electronic device 10 maytransmit “Okay, see you then” 614 that is directly input by the user tothe another electronic device, and display the same on the executionscreen 110 of the message application. For example, the electronicdevice 10 may display a pop-up window for inputting a text in responseto a user's input of selecting an object 612 c that receives the directinput progress. The electronic device 10 may transmit the input textdata to another electronic device through a popup window for inputtingtext and display the text data on the execution screen 110 of themessage application. In this case, the electronic device 10 may alsodisplay an indicator 614 a indicating that “Okay, see you then” 612 hasbeen generated by the user's direct input.

FIG. 7 is a diagram illustrating an example of summarizing anddisplaying a dialogue content by an electronic device using a digitalassistant according to various embodiments.

According to an embodiment, the electronic device 10 may be an AIspeaker. The electronic device 10 may generate the summary voice datausing the voice data which has been received or transmitted from or tothe another electronic device using the digital assistant and providethe same to the user.

According to an embodiment, the summary voice data may include a name ofa counterpart using another electronic device. In addition, the summaryvoice data may include content that needs confirmation of the user. Forexample, the summary voice data may include at least one of a case wherethe confirmation of the user's instruction is necessary, a case where aproposal is included in the text data received from the another device,a case where the voice data which is the same as the case where the usermade a reply after spending a predetermined time on the utterancereceived from the other electronic device is received, and a case wherethe similarity between the text data or voice data received from anotherelectronic device and the text data or voice data previously stored inthe electronic device is less than or equal to a predetermined value.

Referring to FIG. 7, the electronic device 10 may generate and outputsummary voice data, “Would you like to make an appointment at 4 pm asproposed by K?” 710 using the digital assistant. The electronic device10 may receive a user's utterance “tell me more about the conversation”712. The electronic device 10 may output the entire voice data 714transmitted and received with another electronic device using thedigital assistant based on the user's utterance.

According to an embodiment, the electronic device 10 may summarize atleast some content of the voice data or text data which has beentransmitted or received with the another electronic device using thedigital assistant and provide the user with the same, and may guide theuser to grasp main content while saving time.

FIG. 8 is a diagram illustrating an example of guiding by an electronicdevice to a user of another electronic device while waiting forconfirmation of a user using a digital assistant according to anembodiment.

As described above, referring to 3-a of FIG. 3, the electronic device 10may execute a call application to make a call to B hair salon using thedigital assistant, and generate the path rule to proceed a reservationby generating the voice data in response to the dialogue of thecounterpart.

Referring to 8-a of FIG. 8, the electronic device 10 may generate voicedata indicating that a call is being made using the digital assistant atthe time when a call with an another electronic device 800 is started,and transmit the voice data to the another electronic device 800. Forexample, the electronic device 10 may generate voice data “Hello, thisis a digital assistant on behalf of J” 810 and transmit the voice datato the another electronic device 800.

The electronic device 10 may generate the voice data of “Can I make areservation at 3 pm today?” 812 and transmit the voice data to theanother electronic device 800 using the call application.

The electronic device 10 may receive the voice data of “reservation isfull at 3 pm, but 4 pm is available” 814 from the another electronicdevice 800.

According to an embodiment, the electronic device 10 may identify that,using the digital assistant, the content requesting the reservation time4 pm which is different from 3 pm, which is included in thepre-generated voice transmitted to the received voice, as thereservation time. Based on the difference between the reservation timerequested by the user and the reservation time included in the voicereceived from another electronic device, the electronic device 10 maygenerate and output a notification requesting confirmation from the userusing the digital assistant.

The electronic device 10 may generate the voice data requesting wait tothe another electronic device 800 using the digital assistant andtransmit the same. For example, the electronic device 10 may generateand transmit the voice data “Wait a minute. I am confirming with J” 816using the digital assistant.

According to various embodiments, if the time at which the confirmationof the user is received is longer than a predetermined time (forexample, 2 to 3 minutes), the electronic device 10 may generate andtransmit voice data to call back to the another electronic device 800using the digital assistant. For example, the electronic device 10 maygenerate and transmit voice data, “Sorry. I will call you again afterconfirmation” 818 using the digital assistant.

Referring to 8-b of FIG. 8, the electronic device 10 may generate thevoice data “Can I make a reservation at 3 pm today?” 820 and transmitthe voice message to the another electronic device 800 using the callapplication.

The electronic device 10 may receive the voice data “reservation is fullat 3 pm, but 4 pm is available” 822 from the another electronic device800 through the call application.

According to an embodiment, the electronic device 10 may identify that acontent to request a reservation time of 4 pm that is different from 3pm that is included as the reservation time included in a voicepreviously generated and transmitted is included in the received voice.Based on the difference between the reservation time requested by theuser and the reservation time included in the voice received from theanother electronic device 800, the electronic device 10 may generate andoutput a notification requesting confirmation to the user using thedigital assistant.

In addition, the electronic device 10 may generate and transmit voicedata to check whether there is a request for other contents that aredifferent from contents requested by the user to the another electronicdevice 800 using the digital assistant. For example, the electronicdevice 10 may generate voice data, “Wait a minute, I will check with J.Do you have anything else to confirm?” 824 and transmit the voice datato the another electronic device.

According to an embodiment, the electronic device 10 may additionallytransmit the voice data to the another electronic device according totime that is required for a user to respond to the notificationrequesting confirmation of the user.

FIG. 9 is a block diagram illustrating an example process of executing adigital assistant according to an embodiment.

Referring to FIG. 9, a server (for example: the server 20 of FIG. 2) orthe electronic device 10 may include an automatic speech recognition ASRmodule (e.g., including processing circuitry and/or executable programelements) 910, a natural language understanding NLU module (e.g.,including processing circuitry and/or executable program elements) 920,a path planner module (e.g., including processing circuitry and/orexecutable program elements) 930, a dialogue manager DM module (e.g.,including processing circuitry and/or executable program elements) 940,a natural language generator NLG module (e.g., including processingcircuitry and/or executable program elements) 950 and/or a text tospeech TTS module (e.g., including processing circuitry and/orexecutable program elements) 960.

According to an embodiment, the electronic device 10 may store theplurality of above-described modules in a memory (for example: thememory 230 of FIG. 2). The server 20 may store the plurality ofabove-described modules in a data processor (for example: the dataprocessor 260 of FIG. 2).

According to an embodiment, the ASR module 810 may include variousprocessing circuitry and/or executable program elements and convert theuser input to text data.

According to an embodiment, the ASR module 910 may convert the userinput into text data. For example, the ASR module 910 may include aspeech recognition module. The speech recognition module may include anacoustic model and a language model. For example, the acoustic model mayinclude information related to vocalization, and the language model mayinclude information on unit phoneme information and a combination ofunit phoneme information. The speech recognition module may convert theuser's utterance into text data using the information related tovocalization and information on the unit phoneme information.Information about the acoustic model and language model may be storedin, for example, an automatic speech recognition database (ASR DB) 911.

As an embodiment, the NLU module may include various processingcircuitry and/or executable program elements and recognize the intentionof a user by performing syntactic analysis or semantic analysis.Grammatical analysis may divide the user input in grammatical units(e.g., words, phrases, morphemes, or the like), and may grasp whichgrammatical elements the divided units may have. The semantic analysismay be performed using semantic matching, rule matching, formulamatching, or the like. Accordingly, the NLU module may acquire domain,intent, or parameter for expressing the intent by the user input.

As another embodiment, the NLU module 920 may determine user intentionand parameters using the matching rule divided into a domain, anintention, and a parameter (or a slot) for grasping the intention. Forexample, the one domain (e.g., an alarm) may include a plurality ofintents (e.g., alarm setting, alarm cancellation, or the like), and oneintention may include a plurality of parameters (e.g., time, repetitiontimes, alarm sound, or the like). The plurality of rules may include,for example, one or more mandatory element parameters. The matching rulemay be stored in a natural language understanding database (NLU DB) 921.

As a still another embodiment, the NLU module 920 may grasp the meaningof a word extracted from a user input using a linguistic characteristic(e.g., a grammatical element) such as a morpheme or a phrase, anddetermine a user intention by matching the grasped meaning with thedomain and the intention. For example, the NLU module 920 may determinethe user's intention by calculating how many words extracted from userinput are included in each domain and intention.

According to an example embodiment, the NLU module 920 may determine theparameters of the user input using words that become a basis forunderstanding the intent. According to an example embodiment, the NLUmodule 920 may determine the user's intention using the natural languagerecognition database 921 in which the linguistic characteristic forgrasping the intention of the user input is stored.

As still another embodiment, the NLU module 920 may determine a user'sintention using a personal language model (PLM). For example, the NLUmodule 920 may determine the user's intention using personalizedinformation (e.g., contact list, music list). The personalized languagemodel may be stored in, for example, natural language recognitiondatabase 921. According to an embodiment, not only the NLU module 920but also the ASR module 910 may recognize the user's voice using apersonalized language model stored in the natural language recognitiondatabase 921.

As an embodiment, the NLU module 920 may generate a path rule based onthe intent and a parameter of a user input. For example, the NLU module920 may select an application to be executed based on the intention of auser input, and determine an action to be performed in the selectedapplication. The NLU module 920 may generate the path rule bydetermining a parameter corresponding to the determined action.According to an example embodiment, the path rule generated by the NLUmodule may include an application to be executed, an action (e.g., atleast one state) to be executed in the application, and informationabout parameters for executing the action.

As another embodiment, the NLU module 920 may generate one path rule ora plurality of path rules based on the intention and parameter of theuser input. For example, the NLU module 920 may receive a path rule setcorresponding to the first electronic device 10 from a path plannermodule 930, and determine the path rule by mapping the intention and theparameter of the user input to the received path rule set.

As still another embodiment, the NLU module 920 may generate one or aplurality of path rules by determining an application to be executedbased on the intention and parameter of the user input, an action to beexecuted in an application, and a parameter for executing the action.For example, the NLU module 920 may generate the path rule by arrangingthe application to be executed or the action to be executed by theapplication in an ontology or graph model according to the intentionincluded in the voice. The generated path rule may be stored in a pathrule database PR DB 931 through the path planner module 930, forexample. The generated path rule may be added to a path rule set of thepath rule database 931.

As still another embodiment, the NLU module 920 may select at least onepath rule from among a plurality of generated path rules. For example,the NLU module may select an optimal path rule from among the pluralityof path rules. For example, the NLU module 920 may select a plurality ofpath rules when only some actions are specified based on user utterance.The NLU module 920 may determine one path rule among the plurality ofpath rules by an additional input of the user.

According to an embodiment, the path planner module 930 may includevarious processing circuitry and/or executable program elements andselect at least one path rule among a plurality of path rules.

According to an embodiment, the path planner module 930 may transmit apath rule set including a plurality of path rules to the NLU module 920.The plurality of path rules of the path rule set may be stored in a formof a table in a path rule database 231 connected to the path plannermodule 930. The table stored in the path rule database 231 may bestored, for example, by domains or versions of a domain.

According to an embodiment, the path planner module 930 may select onepath rule or a plurality of path rules from the path rule set andtransmit the same to the NLU module 920. For example, the path plannermodule 930 may match the user's intention and parameter with the pathrule set corresponding to the user terminal 100, select one path rule ora plurality of path rules, and transmit the same to the NLU module 920.

According to an embodiment, the path planner module 930 may generate onepath rule or a plurality of path rules using intentions and parametersincluded in the voice. For example, the path planner module 930 maygenerate one path rule or a plurality of path rules by determining anapplication to be executed and an operation to be executed in theapplication based on the intention and the parameters included in thevoice. According to an embodiment, the path planner module 930 may storethe generated path rule in the path rule database 931.

According to an embodiment, the path planner module 930 may store thepath rule generated in the NLU module 920 in the path rule database 931.The generated path rule may be added to the path rule set stored in thepath rule database 931.

According to an embodiment, a table stored in the path rule database 931may include a plurality of path rules or a plurality of path rule sets.The plurality of path rules or the plurality of path rule sets mayreflect a kind, a version, a type, or a characteristic of an apparatusfor performing each path rule.

According to an example embodiment, the dialogue manager DM module 940may include various processing circuitry and/or executable programelements and determine whether the intention included in the voicedetermined by the NLU module 920 is clear. For example, the dialoguemanager module 940 may determine whether the intention included in thevoice is clear based on whether the information of the parameter issufficient. The dialogue manager module 940 may determine whether theparameters identified in the NLU module 920 are sufficient to perform atask. According to an embodiment, when the intention included in thevoice is not clear, the dialogue manager module 940 may perform feedbackfor requesting necessary information to the user. For example, thedialogue manager module 940 may perform feedback requesting informationabout a parameter for determining the user's intention.

According to an example embodiment, the natural language generation NLGmodule 950 may include various processing circuitry and/or executableprogram elements and change the designated information into a text form.The information changed in the form of text may be in a form of naturallanguage speech. The designated information may be, for example,information about an additional input, information for guidingcompletion of an operation corresponding to the user input, orinformation (for example: feedback information for the user input) forguiding an additional input of the user. The information changed in theform of text may be displayed on the display or changed in the form ofvoice.

According to an embodiment, the TTS module 960 may include variousprocessing circuitry and/or executable program elements and change thetext type information to the voice type information. The TTS module 960may receive the text type information from the natural languagegeneration module 950, and change the text format information to thevoice format information. The electronic device 10 may output thechanged voice type information using a speaker, or the like.

According to an embodiment, the NLU module 920, the path planner module930, and the DM module 940 may be implemented as one module. Forexample, the NLU module 920, the path planner module 930, and the DMmodule 940 may be implemented as one module to determine the intentionof the user and the parameter, and generate a reply (for example: pathrule) corresponding to the determined user's intention and theparameter.

As such, the electronic device 10 or the server may implement thedigital assistant using the above-described modules.

FIG. 10 is a diagram illustrating an example learning model may be usedfor determining level of importance to generate a notification of a userby a digital assistant.

According to an example embodiment, the NLU module (for example: the NLUmodule 920 of FIG. 9) may perform learning of at least one learningmodel. For example, the NLU module 920 may include a dialogue importancedetermination model (e.g., including processing circuitry and/orexecutable program elements) 1010. The dialogue importance determinationmodel 1010 may include at least one of, for example, a user instructionconfirmation model (e.g., including processing circuitry and/orexecutable program elements) 1021, a reply type confirmation model(e.g., including processing circuitry and/or executable programelements) 1022, a reply time confirmation model (e.g., includingprocessing circuitry and/or executable program elements) 1023, and a newcontent confirmation model (e.g., including processing circuitry and/orexecutable program elements) 1024.

The user instruction confirmation model 1021, the reply typeconfirmation model 1022, the reply time confirmation model 1023, and thenew content confirmation model 1024 included in the NLU module 920 maybe established in consideration of an applicable field of therecognition model, a purpose of learning, and computer performance of anapparatus, or the like. The user instruction confirmation model 1021,the reply type confirmation model 1022, the reply time confirmationmodel 1023, and the new content confirmation model 1024 included in theNLU module 920 may be, for example, a model based on a neural network.The user instruction confirmation model 1021, the reply typeconfirmation model 1022, the reply time confirmation model 1023, and thenew content confirmation model 1024 included in the NLU module 920 maybe, for example, a neural network model or a deep learning model that isdeveloped from a neural network model.

At least some of the NLU module 920 may be manufactured in the form ofat least one hardware chip and mounted on the electronic device. Forexample, at least some of the NLU module 920 may be manufactured in theform of a dedicated hardware chip for AI, or may be manufactured as apart of a conventional general purpose processor (for example: a centralprocessing unit (CPU) or application processor) or graphics onlyprocessor (for example: graphics processing unit (GPU)), and mounted onthe aforementioned various electronic devices.

According to an embodiment, a dedicated hardware chip for AI may, forexample, be a dedicated processor specialized in probability operation.Having a higher parallel processing performance than the general-useprocessor, the dedicated hardware chip for AI may rapidly process anoperation of AI field such as machine learning.

At least a part of the NLU module 920 may be mounted in one apparatus,or respectively mounted in separate apparatuses. For example, at leastsome of the NLU module 920 may be included in the electronic device 10,and the rest may be included in the server. Alternatively, at least someof the NLU module 920 may be included in a first server, and one may beincluded in a second server that is different from the first server.

At least some of the NLU module 920 may be implemented as a softwaremodule. When at least some of the NLU module 920 is implemented as asoftware module (or a program module including instructions), thesoftware module may be stored in a computer readable non-transitoryreadable recording medium. In this case, at least one software modulemay be provided by an operating system (OS) or by a predeterminedapplication. Some of the at least one software module may be provided byan operating system (OS), and others may be provided by a predeterminedapplication.

FIG. 11 is a diagram illustrating an example function of a userinstruction confirmation model according to an embodiment.

Referring to FIG. 11, the electronic device 10 may be trained to have acriterion to estimate whether the content included in the voice data ortext data generated by the user of the electronic device 10 isconsistent with content included in voice data or text data receivedfrom the another electronic device. The electronic device 10 may betrained to have a criteria regarding which learning data the userinstruction confirmation model 1021 is to use to obtain a criterion forestimating whether the content included in the voice data or text datagenerated by the user or the content included in the voice data or textdata received from the another electronic device, or how the consistencybetween the content included in the voice data or text data generated bythe user using the learning data and the content included in the voicedata or text data received from the another electronic device isestimated.

According to various embodiments, the electronic device 10 may train theuser instruction confirmation model 1021 using a first sentence 1110, asecond sentence 1120 different from the first sentence 1110, andinformation 1130 on a difference between the first sentence and thesecond sentence as learning data. The information 1130 on the differencebetween the first sentence and the second sentence may refer, forexample, to a content that is included in the first sentence 1110 butnot in the second sentence 1120, and a case where the content of thefirst sentence 1110 is opposite to the content of the second sentence1120, or the like. For example, the learning data may be first sentenceof “let's meet at a location A at 3 pm”, a second sentence of “let'smeet at a location B at 4 pm”, and information on the difference betweenthe first sentence and the second sentence such as 3 pm, 4 pm, place A,place B, or the like.

According to various embodiments, the electronic device 10 may estimatewhether the content included in the voice data or text data that aregenerated by the user of the electronic device 10 is consistent with thecontent included in the voice data or text data received from anotherelectronic device based on the learning result.

According to various embodiments, the electronic device 10 may use auser's reply (or feedback) regarding whether the voice data or text datagenerated by the user of the electronic device 10 is supposedlyconsistent with the voice data or text data received from the anotherelectronic device to update the user instruction confirmation model1021.

FIG. 12 is a diagram illustrating an example function of a reply typeconfirmation model according to an embodiment.

Referring to FIG. 12, the electronic device 10 may learn that the replytype confirmation model 1022 has a criterion for estimating whether aninput sentence is a proposal type sentence requesting a specificdecision from a user. The electronic device 10 may be trained to have acriterion regarding which learning data to use in order to acquire thecriteria so that the reply type confirmation model 1022 estimateswhether the input sentence is a proposal type sentence, or how toestimate whether the input sentence is a proposal type sentence usingthe learning data.

According to various embodiments, the electronic device 10 may train thereply type confirmation model 1022 using a sentence 1210 and asentencefinal form 1220 of the proposal type sentence as the trainingdata. The sentencefinal form 1220 of the proposal type sentence is, forexample, “How about˜?”, “Shall we˜?”, “How do you think of ˜?”, “What doyou think of˜?”, or the like.

According to various embodiments, the reply type confirmation model 1022included in the electronic device 10 may estimate whether the voice dataor text data received from the another electronic device is a proposaltype sentence based on the learning result. The reply type confirmationmodel 1022 may derive the estimation result as a score, for example.

According to various embodiments, the electronic device 10 may use theuser's reply regarding the estimation result regarding whether the voicedata or sentence data received from the another electronic device is aproposal type sentence to update the reply type confirmation model 1022.

FIG. 13 is a diagram illustrating an example function of a reply timeconfirmation model according to an embodiment.

Referring to FIG. 13, the electronic device 10 may be trained so thatthe reply time confirmation model 1023 has a criterion for estimating atime required for the user to reply to an input sentence. The electronicdevice 10 may be trained to have a criterion regarding what learningdata the reply time confirmation model 1023 to use to obtain thecriterion for estimating time required till the user replies for theinput sentence, and how to estimate the time required until the userreplies regarding the input sentence using the learning data.

According to various embodiments, the electronic device 10 may train thereply time confirmation model 1023 using the sentence 1310 received fromanother electronic device as learning data and a time 1320 required toreply to the received sentence. According to various embodiments, theelectronic device 10 may use the sentence 1330 answered by the user asthe training data in response to the sentence 1310 received from anotherelectronic device as the training data.

According to various embodiments, the reply time confirmation model 1023included in the electronic device 10 may estimate the time which isexpected to be required until the user responds regarding the voice dataor text data received from the another electronic device based on alearning result. The reply time confirmation model 1023 may derive theestimation result as a score, for example.

According to various embodiments, the electronic device 10 may use theuser's reply regarding the result of estimating the estimated timerequired until the user answers regarding the voice data or text datareceived from the another electronic device to update the reply timeconfirmation model 1023.

FIG. 14 is a diagram illustrating an example function of a reply timeconfirmation model according to an embodiment.

Referring to FIG. 14, the electronic device 10 may learn so that the newcontent confirmation model 1024 has a criterion for estimating whether acontent of an input sentence is unfamiliar in a dialogue with acounterpart. The electronic device 10 may be trained so that the newcontent confirmation model 1024 obtains a criterion for estimatingwhether the content of the input sentence is unfamiliar content in thedialogue with the counterpart and how to estimate whether the content ofthe input sentence is unfamiliar content in the dialogue with thecounterpart using the learning data.

According to various embodiments, the electronic device 10 may train thenew content confirmation model 1024 using dialogue data 1410 with afirst counterpart, dialogue data 1420 with a second counterpart, or thelike, as the learning data.

According to various embodiments, the new content confirmation model1024 included in the electronic device 10 may estimate whether thecontent included in the voice data or text data received from theanother electronic device is an unfamiliar content in a relation withthe counterpart, based on the learning result.

According to various embodiments, the electronic device 10 may use theuser's reply to the result of estimating whether the content included inthe voice data or text data received from the another electronic deviceis an unfamiliar content in the relation with the counterpart to updatethe new content confirmation model 1024.

FIG. 15 is a flowchart illustrating an example of transmitting andreceiving text data or voice data with a third party by an electronicdevice using a digital assistant according to an embodiment.

Referring to operation 1510, the electronic device 10 may execute anapplication to transmit and receive at least one of the text data orvoice data with the another electronic device, in response to occurrenceof at least one event.

The at least one event may refer, for example, to a case where an inputfrom a user is received to execute an application for transmitting andreceiving at least one of text data or voice data with anotherelectronic device. Alternatively, at least one event may refer, forexample, to a case of receiving text data or voice data from anotherelectronic device.

Referring to operation 1520, when at least one of the text data or voicedata is received from the another electronic device, the electronicdevice 10 may generate at least one of the text data or voice data basedon a characteristic of the user's utterance using the digital assistant,and output the data using the application.

For example, the electronic device 10 may acquire the characteristic ofthe user's utterance using at least one of the call log, text inputhistory of the user, and generate the text data reflecting thecharacteristic of the user's utterance using the acquired characteristicof the user's utterance.

Referring to operation 1530, the electronic device 10 may identify thesituation where the user's confirmation is necessary using the digitalassistant in a situation where at least one of the text data or voicedata is generated based on the characteristic of the user's utterance.

The case where confirmation of the user is required, for example, may bethe case that user instruction confirmation is required. That is, somecontents included in the text data or the voice data generated by theuser and some contents of the text data or the voice data received fromanother electronic device may not be consistent.

The case where the user's confirmation is necessary may be, for example,the case where the text data or voice data which the digital assistantshould generate needs to determine whether to accept the proposal. Thatis, when a proposal is included in the text data or voice data receivedfrom another electronic device, a prudential reply may be required, andthe electronic device 10 may identify this case as requiring the user'sconfirmation.

The case where the user's confirmation is required may be, for example,the case where the time of the user for responding to the utteranceneeds to be confirmed. In other words, if the user's time to respond tothe counterpart's voice is longer than the preset time (for example, theuser's average answer possession time), it may be a situation thatprudent reply is necessary. Thus, the electronic device 10 may identifythis case as requiring the user's confirmation.

The case where the user's confirmation is necessary may be, for example,the similarity between the text data or voice data received from theanother electronic device and the text data or voice data prestored inthe electronic device is less than or equal to a predetermined value.The electronic device 10 may be in the case where the prudential replyis necessary, the electronic device 10 may identify this case asrequiring the user's confirmation.

Referring to operation 1540, the electronic device 10 may generate anotification requesting confirmation of the user using the digitalassistant in a situation where the user's confirmation is required, andoutput the notification using the application.

FIG. 16 is a flowchart illustrating an example of transmitting andreceiving text data or voice data with a third party using a digitalassistant by an electronic device and a server according to anembodiment.

Referring to operation 1610, the electronic device 10 may execute anapplication to transmit or receive at least one of the text data orvoice data with the another electronic device, in response to occurrenceof at least one event.

Referring to operation 1620, upon receiving at least one of the textdata or voice data from the another electronic device, the electronicdevice 10 may transmit the data to the server 20.

Referring to operation 1630, upon receiving at least one of the textdata or voice data from the electronic device 10, the server 20 maygenerate at least one of the text data or voice data based on acharacteristic of the user's utterance and transmit the data to theelectronic device 10.

Referring to operation 1640, the server 20 may identify a situationwhere the user's confirmation is necessary using the digital assistant,in a situation where at least one of the text data or voice data isgenerated based on a characteristic of the user's utterance.

Referring to operation 1650, the server 20 may generate a notificationto request confirmation of the user using the digital assistant in asituation where the user's confirmation is necessary, and transmit thenotification to the electronic device 10.

Referring to operation 1660, the electronic device 10 may output thereceived notification using the application.

FIG. 17 is a block diagram of an electronic device 1701 in a networkenvironment 1700 according to various embodiments. An electronic device1701 may include the electronic device 10 of FIG. 1. Referring to FIG.17, in the network environment 1700, the electronic device 1701 maycommunicate with an electronic device 1702 via a first network 1798(e.g., a short-range wireless communication network) or communicate withan electronic device 1704 or a server 1708 via a second network 1799(e.g., wide area network). According to an example embodiment, theelectronic device 1701 may communicate with the electronic device 1704via the server 1708. According to an example embodiment, the electronicdevice 1701 may include a processor 1720, a memory 1730, an input device1750, an audio output device 1755, a display device 1760, an audiomodule 1770, a sensor module 1776, an interface 1777, a haptic module1779, a camera module 1780, a power management module 1788, a battery1789, a communication module 1790 (e.g., a transceiver), a subscriberidentification module 1796, or an antenna module 1797. In someembodiments, at least one (e.g., display device 1760 or camera module1780) of these components may be omitted from electronic device 1701, orone or more other components may be added. In some embodiments, some ofthese components may be implemented as a single integrated circuit. Forexample, the sensor module 1776 (e.g., a fingerprint sensor, an irissensor, or an illumination sensor) may be implemented to be embedded inthe display device 1760 (e.g., a display).

The processor 1720 may control the at least one another component (e.g.,hardware or software component) of the electronic device 1701 connectedto the processor 1720 by executing the software (e.g., the program 1740)and perform various data processing or operation. According to anembodiment, as at least a part of the data processing or operation, theprocessor 1720 may load the command or data received from anothercomponent (e.g., the sensor module 1776 or the communication module1790) to a volatile memory 1732, process command or data stored in thevolatile memory 1732, and store the result data in a non-volatile memory1734. According to an example embodiment, the processor 1720 may includea main processor 1721 (e.g., a central processing unit or an applicationprocessor), and a secondary processor 1723 (e.g., a graphics processingunit, an image signal processor, a sensor hub processor, or acommunication processor) which may be operated together orindependently. Additionally or alternatively, the secondary processor1723 may use less power than the main processor 1721, or may be set tobe specialized to a designated function. The secondary processor 1723may be implemented separately from, or as a part of, the main processor1721.

The secondary processor 1723 may, for example, in place of the mainprocessor 1721 while the main processor 1721 is in an inactive state(for example: sleep) or along with the main processor 1721 while themain processor 1721 is in an active state (for example: execution of anapplication) control a part of the functions or states related to atleast one component (for example: display device 1760, the sensor module1776, or the communication module 1790) among the components of theelectronic device 1701. According to an example embodiment, thesecondary processor 1723 (e.g., an image signal processor or acommunication processor) may be implemented as a part of a functionallyrelated other components (e.g., camera module 1780 or communicationmodule 1790).

The memory 1730 may store various data used by at least one component(e.g., processor 1720 or sensor module 1776) of the electronic device1701. The data may include, for example, software (e.g., program 1740)and input data or output data related with software instructions. Thememory 1730 may include the volatile memory 1732 or non-volatile memory1734.

The program 1740 may be stored in the memory 1730 as software, andinclude, for example, an operating system 1742, middleware 1744, or anapplication 1746.

The input device 1750 may receive a command or data to be used for thecomponents (e.g., processor 1720) of the electronic device 1701 from theoutside (e.g., user) of the electronic device 1701. The input device1750 may include, for example, a microphone, a mouse, or a keyboard.

The sound output device 1755 may output a sound signal to the outside ofthe electronic device 1701. The sound output device 1755 may include,for example, a speaker or a receiver. The speaker may be used forgeneral purposes, such as multimedia playback or recording playback, andthe receiver may be used to receive incoming calls. According to anexample embodiment, the receiver may be implemented separately from, oras a part of, the speaker.

The display device 1760 may visually provide information to an outside(e.g., user) of the electronic device 1701. The display device 1760 mayinclude, for example, a display, a hologram device, a projector, or acontrol circuit for controlling the device. According to an embodiment,the display device 1760 may include a touch circuitry which is set todetect a touch or a sensor circuit (e.g., a pressure sensor) which isset to measure intensity of power generated by the touch.

The audio module 1770 may convert sound into an electric signal, orconvert an electric signal to sound. According to an example embodiment,the audio module 1770 may acquire sound through an input device 1750, oroutput sound through the sound output device 1755, or an externalelectronic device (e.g., electronic device 1702) (e.g., speaker orheadphone) which is directly or wirelessly connected to the electronicdevice 1701.

The sensor module 1776 may detect the operation state of the electronicdevice 1701 (e.g., power or temperature), or an external environmentstate (e.g., a user state), and generate an electrical signal or a datavalue corresponding to the detected state. According to an embodiment,the sensor module 1776 may include, for example, a gesture sensor, agyro sensor, a barometric pressure sensor, a magnetic sensor, anacceleration sensor, a grip sensor, a proximity sensor, a color sensor,an infrared (IR) sensor, a biometric sensor, a temperature sensor, ahumidity sensor, or an illumination sensor.

The interface 1777 may support one or more designated protocols that maybe used by the electronic device 1701 to be connected directly orwirelessly to an external electronic device (e.g., electronic device1702). According to an embodiment, the interface 1777 may include, forexample, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connection terminal 1778 may include a connector through which theelectronic device 1701 may be physically connected to an externalelectronic device (e.g., the electronic device 1702). According to anembodiment, the connection terminal 1778 may include, for example, anHDMI connector, a USB connector, an SD card connector, or an audioconnector (e.g., a headphone connector).

The haptic module 1779 may convert an electrical signal into amechanical stimulus (e.g., vibration or motion) or an electricalstimulus that a user may recognize through a tactile or kinestheticsense. According to an embodiment, the haptic module 1779 may include,for example, a motor, a piezoelectric element, or an electricstimulation device.

The camera module 1780 may photograph a still image or a moving image.According to an embodiment, the camera module 1780 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 1788 may manage power supplied to theelectronic device 1701. According to an embodiment, the power managementmodule 1788 may be implemented as, for example, at least a part of powermanagement integrated circuit (PMIC).

The battery 1789 may supply power to at least one component of theelectronic device 1701. According to an embodiment, the battery 1789 mayinclude, for example, a non-rechargeable primary battery, a rechargeablesecondary battery, or a fuel cell.

The communication module 1790 may support establishment of direct (e.g.:wired) communication channel between the electronic device 1701 and anexternal electronic device (e.g., electronic device 1702, electronicdevice 1704, or server 1708) or wireless communication channel, andcommunication through the established communication channels. Thecommunication module 1790 may include one or more communicationprocessors which are operated independently of the processor 1720 (e.g.,application processor) and support direct (e.g., wired) communication orwireless communication. According to an embodiment, the communicationmodule 1790 may include a wireless communication module 1792 (e.g.,cellular communication module, near field wireless communication module,or global navigation satellite system (GNSS) communication module) orwired communication module 1794 (e.g., local area network (LAN)communication module, or power line communication module). Thecorresponding communication module among these communication modules maycommunicate with an external electronic device via the first network1798 (e.g., Bluetooth, WiFi direct or near field communication networksuch as infrared data association (IrDA)) or the second network 1799(e.g., telecommunication network such as cellular network, Internet, orcomputer network (e.g., LAN or WAN)). These types of communicationmodules may be incorporated into one component (e.g., a single chip) orimplemented with a plurality of components (e.g., a plurality of chips)that are separate from each other. The wireless communication module1792 may confirm and authenticate the electronic device 1701 in thecommunication network such as the first network 1798 or the secondnetwork 1799 using the subscriber information (e.g., internationalmobile subscriber identity (IMSI)) stored in the subscriberidentification module 1796.

The antenna module 1797 may transmit a signal or power to an externaldevice (e.g., an external electronic device) or receive a signal orpower from the outside. According to an example embodiment, the antennamodule 1797 may include one or more antennas from which at least oneantenna suitable for a communication method used in a communicationnetwork, such as the first network 1798 or the second network 1799, maybe selected by, for example, the communication module 1790. A signal orpower may be transmitted or received between the communication module1790 and the external electronic device via the selected at least oneantenna.

At least a part of the components may be interconnected through thecommunication method (e.g., bus, general purpose input and output(GPIO), serial peripheral interface (SPI), or mobile industry processorinterface (MIPI)) among peripheral devices and exchange a signal (e.g.,command or data) from each other.

According to an example embodiment, the command or data may betransmitted or received between the electronic device 1701 and theexternal electronic device 1704 via the server 1708 connected to thesecond network 1799. Each of the electronic devices 1702 and 1704 may bedevices which are the same or different types from the electronic device1701. For example, when the electronic device 1701 is a personalelectronic device for an individual such as smartphone, externalelectronic devices 1702 and 1704 may be a mobile robot TV or afixed-type common use electronic device for a plurality of people suchas or a refrigerator. According to an embodiment, whole or a part of theoperations executed by the electronic device 1701 may be executed by oneor more external devices among the external electronic devices 1702,1704, or 1708. For example, when the electronic device 1701 has toperform a function or service automatically, or in response to a requestfrom a user or another device, the electronic device 1701 may requestone or more external electronic devices to perform at least a part ofthe function or the service instead of, or in addition to, performingthe function or service by itself. The one or more external electronicdevices that have received the request may execute at least a portion ofthe requested function or service, or an additional function or serviceassociated with the request, and transmit the result of the execution tothe electronic device 1701. The electronic device 1701 may process theresult as is or additionally, and provide the result as at least aportion of the response to the request. For this purpose, for example,cloud computing, distributed computing, or client-server computingtechnology may be used.

For example, when the electronic device 1701 receives the user voice andtransmits the voice to external electronic devices 1702 and 1704, theexternal electronic devices 1702 and 1704 may execute a functioncorresponding to the user's voice by analyzing the received voice andgenerating the path rule.

The various embodiments of the disclosure may be implemented withsoftware (e.g., program 1740) including one or more instructions storedin the storage medium (e.g., internal memory 1736 or external memory1738) readable by a machine (e.g., electronic device 1701). For example,the processor (e.g., processor 1720) of a device (e.g., electronicdevice 1701) may call at least one instruction among one or moreinstructions stored in the storage medium and execute the instructions.This enables a device to be operated to perform at least one functionaccording to the called at least one instructions. The instructions mayinclude a code generated by a compiler or a code executable by aninterpreter. The storage medium readable by a machine may be provided inthe form of a non-transitory storage medium that is a tangible deviceand may not include a signal (e.g., electromagnetic wave). This termdoes not distinguish that data is permanently or temporarily stored inthe storage medium.

According to an embodiment, the method according to various embodimentsdisclosed herein may be provided in a computer program product. Acomputer program product may be exchanged between a seller and apurchaser as a commodity. A computer program product may be distributedin the form of a machine-readable storage medium (e.g., compact discread only memory (CD-ROM)) or distributed online through an applicationstore (e.g. PlayStore™) directly between two user devices (e.g.,smartphones). In the case of on-line distribution, at least a portion ofthe computer program product may be stored temporarily or at leasttemporarily in a storage medium such as a manufacturer's server, aserver of an application store, or a memory of a relay server.

Each of the elements (for example, a module or a program) according tovarious embodiments may be composed of a single entity or a plurality ofentities, and some sub-elements of the abovementioned sub-elements maybe omitted. The elements may be further included in various embodiments.Alternatively or additionally, some elements (e.g., modules or programs)may be integrated into one entity to perform the same or similarfunctions performed by each respective element prior to integration.Operations performed by a module, program, or other element, inaccordance with various embodiments, may be performed sequentially, in aparallel, repetitive, or heuristically manner, or at least someoperations may be performed in a different order, or at least one otheroperations may be added.

While various example embodiments of the disclosure have beenillustrated and described herein, it will be understood that the variousexample embodiments are intended to be illustrative, not limiting. Oneof ordinary skill in the art will understand that various changes inform and details may be made without departing from the true spirit andfull scope of the disclosure as set forth, for example, in the appendedclaims and their equivalents.

What is claimed is:
 1. An electronic device comprising: a communicationmodule comprising communication circuitry; a processor; and a memorystoring at least one command which, when executed by the processor,is/are configured to control the electronic device to: in response tooccurrence of at least one event, execute an application configured totransmit and receive at least one of text data or voice data withanother electronic device using the communication module, receive textdata or voice data from the another electronic device, identifysimilarity of at least some content of the text data or voice datareceived from another electronic device and text data or voice dataprestored in the electronic device is less than or equal to apredetermined value, identify, using a digital assistant, whether auser's confirmation is necessary for a response based on theidentification that the similarity of at least some content of the textdata or voice data received from another electronic device and text dataor voice data prestored in the electronic device being less than orequal to the predetermined value, the response to be generated based ona characteristic of the user's utterance, based on identification thatthe user's confirmation is necessary, provide a notification includingan object to approve content of the received text data or voice data;and based on identification that the user's confirmation is unnecessary,transmit to the another electronic device the response to the anotherelectronic device without providing the notification.
 2. The electronicdevice of claim 1, wherein the notification to request the confirmationcomprises text data or voice data based on text data or voice datareceived from the another electronic device.
 3. The electronic device ofclaim 2, wherein the electronic device further comprises a display, andwherein the at least one command, when executed by the processor,controls the electronic device to: display at least one of an object toapprove content of text data included in the notification to request theconfirmation, an object to reject, and an object to receive a directinput.
 4. The electronic device of claim 1, wherein the at least onecommand, when executed by the processor, controls the electronic deviceto: identify that the confirmation is required based on at least somecontent of text data or voice data received from the another electronicdevice not being consistent with at least some content of input textdata or voice data.
 5. The electronic device of claim 1, wherein the atleast one command, when executed by the processor, controls theelectronic device to: identify that the confirmation is required basedon a proposal included in text data or voice data received from theanother electronic device.
 6. The electronic device of claim 1, whereinthe at least one command, when executed by the processor, controls theelectronic device to: identify that the confirmation is required basedon a time that is required for responding to text data or voice datathat are same or similar to the text data or voice data received fromthe another electronic device is greater than or equal to apredetermined time.
 7. The electronic device of claim 1, wherein the atleast one command, when executed by the processor, controls theelectronic device to: identify that the confirmation is required basedon a similarity between at least some of content included in text dataor voice data received from the another electronic device and text dataor voice data pre stored in the electronic device being less than orequal to a predetermined value.
 8. The electronic device of claim 1,wherein the at least one command, when executed by the processor,controls the electronic device to: transmit, to the another electronicdevice, sound data or an indicator to provide a notification that thedigital assistant generates at least one of the text data or voice databased on a characteristic of the utterance and that the text data orvoice data is generated by the digital assistant based on outputting thedata using the application.
 9. The electronic device of claim 1, whereinthe at least one command, when executed by the processor, controls theelectronic device to: generate summary text data or summary voice datausing at least a portion of the text data or voice data received fromthe another electronic device or the text data or voice data generatedby the digital assistant, and output the data using the application. 10.A server comprising: a data acquisition unit comprising data acquisitioncircuitry configured to acquire text data or voice data from an externalelectronic device; a data processor configured to: identify whether auser's confirmation is required for a response based on anidentification that similarity of at least some content of the text dataor voice data received from another electronic device and text data orvoice data prestored in the external electronic device being less thanor equal to a predetermined value, wherein the response is to begenerated based on a characteristic of the user's utterance, provide anotification to request the user's confirmation based on the user'sconfirmation being required; and a data outputter, comprising dataoutput circuitry, configured to transmit a notification requesting theuser's confirmation to the external electronic device, wherein thenotification includes an object to approve content of the acquired textdata or voice data.
 11. A method of controlling an electronic device,the method comprising: in response to occurrence of at least one event,executing an application for transmitting and receiving at least one oftext data or voice data with another electronic device; receiving textdata or voice data from the another electronic device; identifying thatsimilarity of at least some content of the text data or voice datareceived from another electronic device and text data or voice dataprestored in the electronic device is less than or equal to apredetermined value identifying, using a digital assistant, whether auser's confirmation is necessary for a response based on theidentification that the similarity of at least some content of the textdata or voice data received from another electronic device and text dataor voice data prestored in the electronic device being less than orequal to a predetermined value, wherein the response being generatedbased on a characteristic of the user's utterance, based on theidentification that the user's confirmation being necessary, providing anotification including an object to approve content of the received textdata or voice data; and based on the identification that the user'sconfirmation being unnecessary, transmitting the response to the anotherelectronic device without providing the notification.
 12. The method ofclaim 11, wherein the notification to request confirmation comprisestext data or voice data based on text data or voice data received fromthe another electronic device.
 13. The method of claim 12, furthercomprising: displaying at least one of an object to approve content oftext data included in the notification to request the confirmation, anobject to reject, and an object to receive a direct input based onconfirmation being necessary.
 14. The method of claim 11, furthercomprising: identifying that the confirmation is required based on atleast some content of text data or voice data received from the anotherelectronic device not being consistent with at least some content oftext data or voice data that are input.
 15. The method of claim 11,further comprising: identifying that the confirmation is required basedon a proposal being included in text data or voice data received fromthe another electronic device.
 16. The method of claim 11, furthercomprising: identifying that the confirmation is required based on atime required for a user to respond to text data or voice data that aresame or similar to the text data or voice data received from the anotherelectronic device being greater than or equal to a predetermined time.17. The method of claim 11, further comprising: identifying that theconfirmation is required based on a similarity between at least some ofcontent included in text data or voice data received from the anotherelectronic device and text data or voice data prestored in theelectronic device being less than or equal to a predetermined value. 18.The method of claim 11, further comprising: generating at least one ofthe text data or voice data by the digital assistant based on acharacteristic of the utterance and transmitting, to the anotherelectronic device, sound data or an indicator to provide a notificationthat the text data or voice data is generated by the digital assistant.19. The method of claim 11, further comprising: generating summary textdata or summary voice data using at least a portion of the text data orvoice data received from the another electronic device or the text dataor voice data generated by the digital assistant, and outputting thedata using the application.